Creating a corpus for automatic punctuation prediction in Persian texts

We present a novel corpus for automatic punctuation prediction in Persian texts. Punctuation prediction is an important task in automatic speech recognition (ASR). The output of ASR systems is typically a raw sequence of words with no punctuation marks; this makes the text difficult or even impossible to make sense of for humans and also for any text processing unit. In this work, we have assembled a state-of-the-art Persian corpus to train and test a punctuation prediction model. To the best of our knowledge, this is the first ever corpus specifically designed for punctuation prediction in Persian texts. The corpus is a modification of a manually part-of-speech (POS) tagged Persian one, with almost 2.6 million words, including punctuation marks. We have made many diligent improvements to the already existing corpus to make one that deliberately facilitates experimental studies on Persian punctuation prediction: 1- replacing 3175 word types with their correct form, 2- normalizing the words (e.g. replacing kashida with hyphen), 3- correcting 451 and 192 words with incorrect DELM and DEFAULT tags, respectively, 4- investigating 17 word types to correct the punctuations around them, and 5- making numerous corrections to the punctuation marks. The final corpus contains nearly 2.3 million words and 221 thousand punctuation marks. Finally, we have trained and tested a CRF (conditional random field) model that shows a micro-averaged F1-score of 60.69% in our preliminary experiments.